{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Agent, action converters and l2rpn_baselines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is recommended to have a look at the [0_basic_functionalities](0_basic_functionalities.ipynb), [1_Observation_Agents](1_Observation_Agents.ipynb) and [2_Action_GridManipulation](2_Action_GridManipulation.ipynb) notebooks before getting into this one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Objectives**\n",
    "\n",
    "In this notebook we will expose :\n",
    "* how to use the \"converters\": these allow to link several different representations of the actions (for example as `Action` objects or integers).\n",
    "* how to train a (naive) Agent using reinforcement learning.\n",
    "* how to inspect (rapidly) the action taken by the Agent.\n",
    "\n",
    "**NB** In this tutorial, we train an Agent inspired from this blog post: [deep-reinforcement-learning-tutorial-with-open-ai-gym](https://towardsdatascience.com/deep-reinforcement-learning-tutorial-with-open-ai-gym-c0de4471f368). Many other different reinforcement learning tutorials exist. The code presented in this notebook only aims at demonstrating how to use the Grid2Op functionalities to train a Deep Reinforcement learning Agent and inspect its behaviour, but not at building a very smart agent. Nothing about the performance, training strategy, type of Agent, meta parameters, etc, should be retained as a common practice.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import grid2op"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"my_id_menu_nb\">run previous cell, wait for 2 seconds</div>\n",
       "<script>\n",
       "function repeat_indent_string(n){\n",
       "    var a = \"\" ;\n",
       "    for ( ; n > 0 ; --n)\n",
       "        a += \"    \";\n",
       "    return a;\n",
       "}\n",
       "// look up into all sections and builds an automated menu //\n",
       "var update_menu_string = function(begin, lfirst, llast, sformat, send, keep_item, begin_format, end_format) {\n",
       "    var anchors = document.getElementsByClassName(\"section\");\n",
       "    if (anchors.length == 0) {\n",
       "        anchors = document.getElementsByClassName(\"text_cell_render rendered_html\");\n",
       "    }\n",
       "    var i,t;\n",
       "    var text_menu = begin;\n",
       "    var text_memo = \"<pre>\\nlength:\" + anchors.length + \"\\n\";\n",
       "    var ind = \"\";\n",
       "    var memo_level = 1;\n",
       "    var href;\n",
       "    var tags = [];\n",
       "    var main_item = 0;\n",
       "    var format_open = 0;\n",
       "    for (i = 0; i <= llast; i++)\n",
       "        tags.push(\"h\" + i);\n",
       "\n",
       "    for (i = 0; i < anchors.length; i++) {\n",
       "        text_memo += \"**\" + anchors[i].id + \"--\\n\";\n",
       "\n",
       "        var child = null;\n",
       "        for(t = 0; t < tags.length; t++) {\n",
       "            var r = anchors[i].getElementsByTagName(tags[t]);\n",
       "            if (r.length > 0) {\n",
       "child = r[0];\n",
       "break;\n",
       "            }\n",
       "        }\n",
       "        if (child == null) {\n",
       "            text_memo += \"null\\n\";\n",
       "            continue;\n",
       "        }\n",
       "        if (anchors[i].hasAttribute(\"id\")) {\n",
       "            // when converted in RST\n",
       "            href = anchors[i].id;\n",
       "            text_memo += \"#1-\" + href;\n",
       "            // passer à child suivant (le chercher)\n",
       "        }\n",
       "        else if (child.hasAttribute(\"id\")) {\n",
       "            // in a notebook\n",
       "            href = child.id;\n",
       "            text_memo += \"#2-\" + href;\n",
       "        }\n",
       "        else {\n",
       "            text_memo += \"#3-\" + \"*\" + \"\\n\";\n",
       "            continue;\n",
       "        }\n",
       "        var title = child.textContent;\n",
       "        var level = parseInt(child.tagName.substring(1,2));\n",
       "\n",
       "        text_memo += \"--\" + level + \"?\" + lfirst + \"--\" + title + \"\\n\";\n",
       "\n",
       "        if ((level < lfirst) || (level > llast)) {\n",
       "            continue ;\n",
       "        }\n",
       "        if (title.endsWith('¶')) {\n",
       "            title = title.substring(0,title.length-1).replace(\"<\", \"&lt;\")\n",
       "         .replace(\">\", \"&gt;\").replace(\"&\", \"&amp;\");\n",
       "        }\n",
       "        if (title.length == 0) {\n",
       "            continue;\n",
       "        }\n",
       "\n",
       "        while (level < memo_level) {\n",
       "            text_menu += end_format + \"</ul>\\n\";\n",
       "            format_open -= 1;\n",
       "            memo_level -= 1;\n",
       "        }\n",
       "        if (level == lfirst) {\n",
       "            main_item += 1;\n",
       "        }\n",
       "        if (keep_item != -1 && main_item != keep_item + 1) {\n",
       "            // alert(main_item + \" - \" + level + \" - \" + keep_item);\n",
       "            continue;\n",
       "        }\n",
       "        while (level > memo_level) {\n",
       "            text_menu += \"<ul>\\n\";\n",
       "            memo_level += 1;\n",
       "        }\n",
       "        text_menu += repeat_indent_string(level-2);\n",
       "        text_menu += begin_format + sformat.replace(\"__HREF__\", href).replace(\"__TITLE__\", title);\n",
       "        format_open += 1;\n",
       "    }\n",
       "    while (1 < memo_level) {\n",
       "        text_menu += end_format + \"</ul>\\n\";\n",
       "        memo_level -= 1;\n",
       "        format_open -= 1;\n",
       "    }\n",
       "    text_menu += send;\n",
       "    //text_menu += \"\\n\" + text_memo;\n",
       "\n",
       "    while (format_open > 0) {\n",
       "        text_menu += end_format;\n",
       "        format_open -= 1;\n",
       "    }\n",
       "    return text_menu;\n",
       "};\n",
       "var update_menu = function() {\n",
       "    var sbegin = \"\";\n",
       "    var sformat = '<a href=\"#__HREF__\">__TITLE__</a>';\n",
       "    var send = \"\";\n",
       "    var begin_format = '<li>';\n",
       "    var end_format = '</li>';\n",
       "    var keep_item = -1;\n",
       "    var text_menu = update_menu_string(sbegin, 2, 4, sformat, send, keep_item,\n",
       "       begin_format, end_format);\n",
       "    var menu = document.getElementById(\"my_id_menu_nb\");\n",
       "    menu.innerHTML=text_menu;\n",
       "};\n",
       "window.setTimeout(update_menu,2000);\n",
       "            </script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = None\n",
    "try:\n",
    "    from jyquickhelper import add_notebook_menu\n",
    "    res = add_notebook_menu()\n",
    "except ModuleNotFoundError:\n",
    "    print(\"Impossible to automatically add a menu / table of content to this notebook.\\nYou can download \\\"jyquickhelper\\\" package with: \\n\\\"pip install jyquickhelper\\\"\")\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I) Manipulating action representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Grid2op package has been built with an \"object-oriented\" perspective: almost everything is encapsulated in a dedicated `class`. This allows for more customization of the plateform.\n",
    "\n",
    "The downside of this approach is that machine learning methods, especially in deep learning, often prefer to deal with vectors rather than with \"complex\" objects. Indeed, as we covered in the previous tutorials on the platform, we saw that building our own actions can be tedious and can sometime require important knowledge of the powergrid.\n",
    "\n",
    "On the contrary, in most of the standard Reinforcement Learning environments, actions have a higher representation. For example in pacman, there are 4 different types of actions: turn left, turn right, go up and do down. This allows for easy sampling (if you need to achieve an uniform sampling, you simply need to randomly pick a number between 0 and 3 included) and an easy representation: each action can be represented as a different component of a vector of dimension 4 [because there are 4 actions]. \n",
    "\n",
    "On the other hand, this representation is not \"human friendly\". It is quite convenient in the case of pacman because the action space is rather small, making it possible to remember which action corresponds to which component, but in the case of the grid2op package, there are hundreds or even thousands of actions, making it impossible to remember which component corresponds to which action. We suppose that we do not really care about this here, as tutorials on Reinforcement Learning with discrete action space often assume that actions are labeled with integers (such as in pacman for example).\n",
    "\n",
    "However, to allow RL agent to train more easily, we allow to make some \"[Converters](https://grid2op.readthedocs.io/en/latest/converters.html)\" whose roles are to allow an agent to deal with a custom representation of the action space. The class [AgentWithConverter](https://grid2op.readthedocs.io/en/latest/agent.html#grid2op.Agent.AgentWithConverter) is perfect for such usage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/benjamin/Documents/grid2op_dev/getting_started/grid2op/MakeEnv/Make.py:248: UserWarning: You are using a development environment. This environment is not intended for training agents.\n",
      "  warnings.warn(_MAKE_DEV_ENV_WARN)\n"
     ]
    }
   ],
   "source": [
    "# import the usefull class\n",
    "import numpy as np\n",
    "\n",
    "from grid2op import make\n",
    "from grid2op.Agent import RandomAgent \n",
    "max_iter = 100 # to make computation much faster we will only consider 50 time steps instead of 287\n",
    "train_iter = 1000\n",
    "env_name = \"rte_case14_redisp\"\n",
    "env = make(env_name, test=True)\n",
    "env.seed(0)  # this is to ensure all sources of randomness in the environment are reproducible\n",
    "my_agent =  RandomAgent(env.action_space)\n",
    "my_agent.seed(0)  # this is to ensure that all actions made by this random agent will be the same"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it. This agent will be able to perform any action, but instead of going through the description of the actions from a powersystem point of view (ie setting what is connected to what, what is disconnected etc.) it will simply choose an integer with the method `my_act`. This integer will then be converted back to a proper action.\n",
    "\n",
    "Here is an example of the action representation as seen by the Agent (here, integers):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "172\n",
      "47\n",
      "117\n"
     ]
    }
   ],
   "source": [
    "for el in range(3):\n",
    "    print(my_agent.my_act(None, None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below you can see that the `act` function behaves as expected, handling proper `Action` objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element:\n",
      "\t \t - assign bus 2 to line (extremity) 1 [on substation 4]\n",
      "\t \t - assign bus 2 to line (extremity) 9 [on substation 4]\n",
      "\t \t - assign bus 1 to line (extremity) 11 [on substation 4]\n",
      "\t \t - assign bus 2 to line (origin) 17 [on substation 4]\n",
      "\t \t - assign bus 2 to load 4 [on substation 4]\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - Change the bus of the following element:\n",
      "\t \t - switch bus of line (origin) 11 [on substation 3]\n",
      "\t - NOT force any particular bus configuration\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - Set the bus of the following element:\n",
      "\t \t - assign bus 2 to line (origin) 2 [on substation 8]\n",
      "\t \t - assign bus 1 to line (origin) 3 [on substation 8]\n",
      "\t \t - assign bus 1 to line (extremity) 16 [on substation 8]\n",
      "\t \t - assign bus 1 to line (extremity) 19 [on substation 8]\n",
      "\t \t - assign bus 1 to load 6 [on substation 8]\n"
     ]
    }
   ],
   "source": [
    "for el in range(3):\n",
    "    print(my_agent.act(None, None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB** lots of these actions are equivalent to the \"do nothing\" action at some point. For example, trying to reconnect a powerline that is already connected will not do anything. The same for topology. If everything is already connected to bus 1, then the action to connect things to bus 1 on the same substation will not affect the powergrid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II) Training an Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will show how to build a Q-learning Agent. Most of the code originated from this blog post (which was deleted) [https://towardsdatascience.com/deep-reinforcement-learning-tutorial-with-open-ai-gym-c0de4471f368](https://towardsdatascience.com/deep-reinforcement-learning-tutorial-with-open-ai-gym-c0de4471f368). \n",
    "\n",
    "The goal of this notebook is to demonstrate how to train an agent using grid2op framework. The key message is: since grid2op fully implements the gym API, it is rather easy to do. We will use the [l2rpn baselines](https://github.com/rte-france/l2rpn-baselines) repository and implement a Double Dueling Deep Q learning Algorithm. For more information, you can look for the code in the dedicated repository [here](https://github.com/rte-france/l2rpn-baselines/tree/master/l2rpn_baselines/DoubleDuelingDQN).\n",
    "\n",
    "**Requirements** This notebook requires having `keras` installed on your machine as well as the `l2rpn_baselines` repository.\n",
    "\n",
    "As always in these notebooks, we will use the `rte_case14_realistic` test Environment. More data is available if you don't pass the `test=True` parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II.A) Defining some \"helpers\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The type of Agent we are using requires a bit of setup, independantly of Grid2Op. We will reuse the code shown in \n",
    "[https://towardsdatascience.com/deep-reinforcement-learning-tutorial-with-open-ai-gym-c0de4471f368](https://towardsdatascience.com/deep-reinforcement-learning-tutorial-with-open-ai-gym-c0de4471f368) and in [Reinforcement-Learning-Tutorial](https://github.com/abhinavsagar/Reinforcement-Learning-Tutorial) from Abhinav Sagar. The code is registered under the *MIT license* found here: [MIT License](https://github.com/abhinavsagar/Reinforcement-Learning-Tutorial/blob/master/LICENSE).\n",
    "\n",
    "This first section aims at defining these classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to install the l2rpn_baselines library. Since this library is uploaded on Pypi this can be done easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To install l2rpn_baselines, either uncomment the cell below, or type, in a command prompt:\n",
      "\t/usr/bin/python3 -m pip install l2rpn_baselines\n"
     ]
    }
   ],
   "source": [
    "print(\"To install l2rpn_baselines, either uncomment the cell below, or type, in a command prompt:\\n{}\".format(\n",
    "    (\"\\t{} -m pip install l2rpn_baselines\".format(sys.executable))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !$sys.executable -m pip install l2rpn_baselines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But first let's import the necessary dependencies :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#tf2.0 friendly\n",
    "import numpy as np\n",
    "import random\n",
    "import warnings\n",
    "import l2rpn_baselines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II.B) Adaptation of the inputs\n",
    "\n",
    "For many of the Deep Reinforcement Learning problems (for example for model used to play Atari games), the inputs are images and the outputs are integers that encode the different possible actions (typically \"move up\" or \"move down\" in Atari). In our system (the powergrid) it is rather different. We did our best to make the convertion between complex and simple structures easy. Indeed, the use of converters such as ([IdToAct](https://grid2op.readthedocs.io/en/latest/converter.html#grid2op.Converter.IdToAct)) allows easily to:\n",
    "- convert the class \"Observation\" into vectors automatically\n",
    "- map the actions from integers to complete `Action` objects defined in the previous notebooks\n",
    "\n",
    "In essence, a converter allows to manipulate the \"action space\" of the Agent and is such that:\n",
    "- the Agent manipulates a simple, custom structure of actions\n",
    "- the Converter takes care of mapping from this simple structure to complex grid2op Action / Observation objects\n",
    "- as a whole, the Agent will actually return full Action / Observation objects to the environment while only working with more simple objects\n",
    "\n",
    "\n",
    "#### A note on the converter\n",
    "To use this converter, the Agent must inherit from the class [`grid2op.Agent.AgentWithConverter`](https://grid2op.readthedocs.io/en/latest/agent.html#grid2op.Agent.AgentWithConverter) and implement the following interface (shown here as an example):\n",
    "\n",
    "\n",
    "```python\n",
    "from grid2op.Agent import AgentWithConverter\n",
    "class MyAgent(AgentWithConverter):\n",
    "    def __init__(self, action_space, action_space_converter=None):\n",
    "        super(MyAgent, self).__init__(action_space=action_space, action_space_converter=action_space_converter)\n",
    "        # for example you can define here all the actions you will consider\n",
    "        self.my_actions = [action_space(),\n",
    "                           action_space({\"redispatching\": [0,+1]}),\n",
    "                           action_space({\"set_line_status\": [(0,-1)]}),\n",
    "                           action_space({\"change_bus\": {\"lines_or_id\": [12]}}),\n",
    "                          ...\n",
    "                          ]\n",
    "        # or load them from a file for example...\n",
    "        # self.my_action = np.load(\"my_action_pre_selected.npy\")\n",
    "        \n",
    "        # you can also in this agent load a neural network...\n",
    "        self.my_nn_model = model.load(\"my_saved_neural_network_weights.h5\")\n",
    "    \n",
    "    def convert_obs(self, observation):\n",
    "        \"\"\"\n",
    "        This method is used to convert the observation, represented as a class Observation in input\n",
    "        into a \"transformed_observation\" that will be manipulated by the agent\n",
    "        An example here will transform the observation into a numpy array.\n",
    "        \n",
    "        It is recommended to modify it to suit your needs.\n",
    "        \n",
    "        \"\"\"\n",
    "        return observation.to_vect()\n",
    "    \n",
    "    def convert_act(self, encoded_act):\n",
    "        \"\"\"\n",
    "        This method will take an \"encoded_act\" (for example a integer) into a valid grid2op action.\n",
    "        \"\"\"\n",
    "        if encoded_act < 0 or encoded_act > len(self.my_action):\n",
    "            raise RuntimeError(\"Invalid action with id {}\".format(encoded_act))\n",
    "        return self.my_actions[encoded_act]\n",
    "    \n",
    "    def my_act(self, transformed_observation, reward, done=False):\n",
    "        \"\"\"\n",
    "        This is the main function where you can take your decision.\n",
    "        \n",
    "        Instead of:\n",
    "        - calling \"act(observation, reward, done)\" you implement \n",
    "          \"my_act(transformed_observation, reward, done)\"\n",
    "        - this manipulates only \"transformed_observation\" fully flexible as you defined \"convert_obs\"\n",
    "        - and returns \"encoded_action\" that are then digest automatically by \n",
    "          \"convert_act(encoded_act)\" and to return valid actions.\n",
    "        \n",
    "        Here we suppose, as many dqn agent, that `my_nn_model` return a vector of size \n",
    "        nb_actions filled with number between 0 and 1 and we take the action given the highest score\n",
    "        \"\"\"\n",
    "        pred_score = self.my_nn_model.predict(transformed_observation, reward, done)\n",
    "        res = np.argmax(pred_score)\n",
    "        return res\n",
    "```\n",
    "And that's it. There is nothing else to do, your agent is ready to learn how to control a powergrid using only these 3 functions.\n",
    "\n",
    "\n",
    "**NB** A few things are worth noting:\n",
    "- if you use an agent with a converter, do not modify the method **act** but rather change the method **my_act**, this is really important !\n",
    "- the method **my_act**, which you will have to implement, takes a transformed observation as an argument and must return a transformed action. This transformed action will then be converted back to the original action space to an actual `Action` object in the **act** method, that you must leave unchanged.\n",
    "- some automatic functions can compute the set of all possible actions, so there is no need to write \"self.my_actions = ...\". This was done as an example only.\n",
    "- if the converter is properly set up, you don't even need to modify \"convert_obs(self, observation)\" and \"convert_act(self, encoded_act)\" as default convertions are already provided in the implementation. However, if you want to work with a custom observation space and action space, you can modify these two methods in order to have the convertions that you need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we consider the observation as a whole and do not try any modifications of the features. This means that the vector (the observation) that the agent will receive is going to be really big, not scaled and filled with a lot of information that may not be really useful. It could be tried to select only a subset of the available features and apply a pre-processing function to them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II.C) Writing the code of the Agent and train it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### a) Code of the agent\n",
    "\n",
    "Here we show the most interesting part (in this tutorial) of the code that is implemented as a baseline. For a full description of the code, you can go take a look [here](https://github.com/rte-france/l2rpn-baselines/tree/master/l2rpn_baselines/DoubleDuelingDQN).\n",
    "\n",
    "This is the `DoubleDuelingDQN_NN.py` file:\n",
    "\n",
    "```python\n",
    "import tensorflow.keras as tfk\n",
    "class DoubleDuelingDQN_NN(object):\n",
    "    \"\"\"Constructs the desired deep q learning network\"\"\"\n",
    "    def __init__(self,\n",
    "                 action_size,\n",
    "                 observation_size,                 \n",
    "                 HIDDEN_FOR_SIMPLICITY\n",
    "                ):\n",
    "        self.action_size = action_size\n",
    "        self.observation_size = observation_size\n",
    "        HIDDEN_FOR_SIMPLICITY\n",
    "\n",
    "    def construct_q_network(self):\n",
    "        \"\"\"\n",
    "        we showed this here to tell you it was exactly like any keras implementation\n",
    "        \"\"\"\n",
    "        input_layer = tfk.Input(shape = (self.observation_size * self.num_frames,), name=\"input_obs\")\n",
    "        lay1 = tfkl.Dense(self.observation_size * 2, name=\"fc_1\")(input_layer)\n",
    "        lay1 = tfka.relu(lay1, alpha=0.01) #leaky_relu\n",
    "        ...\n",
    "        HIDDEN_FOR_SIMPLICITY\n",
    "        ...\n",
    "        self.model = tfk.Model(...)\n",
    "\n",
    "    def random_move(self):\n",
    "        \"\"\"\n",
    "        Moves are encoded by a random number between 0 and the total number of actions.\n",
    "        Easy to do a random move isn't it ? :-)\n",
    "        \"\"\"\n",
    "        opt_policy = np.random.randint(0, self.action_size)\n",
    "        return opt_policy\n",
    "        \n",
    "    def predict_move(self, data):\n",
    "        \"\"\"\n",
    "        in this example we decided to show \n",
    "        \"\"\"\n",
    "        model_input = data.reshape(1, self.observation_size * self.num_frames)\n",
    "        q_actions = self.model.predict(model_input, batch_size = 1)     \n",
    "        opt_policy = np.argmax(q_actions)\n",
    "        return opt_policy, q_actions[0]\n",
    "```\n",
    "\n",
    "This is the `DoubleDuelingDQN.py` file:\n",
    "```python\n",
    "from grid2op.Agent import AgentWithConverter  # all converter agent should inherit this\n",
    "from grid2op.Converter import IdToAct  # this is the automatic converter to convert action given as ID (integer)\n",
    "# to valid grid2op action (in particular it is able to compute all actions).\n",
    "\n",
    "from l2rpn_baselines.DoubleDuelingDQN.DoubleDuelingDQN_NN import DoubleDuelingDQN_NN\n",
    "class DoubleDuelingDQN(AgentWithConverter):\n",
    "    def __init__(self,\n",
    "                 observation_space,\n",
    "                 action_space,\n",
    "                 HIDDEN_FOR_SIMPLICITY\n",
    "                ):\n",
    "        ...\n",
    "        HIDDEN_FOR_SIMPLICITY\n",
    "        ...\n",
    "        # Load network graph\n",
    "        self.Qmain = DoubleDuelingDQN_NN(self.action_size,\n",
    "                                         self.observation_size,\n",
    "                                         HIDDEN_FOR_SIMPLICITY)\n",
    "    ## Agent Interface\n",
    "    def convert_obs(self, observation):\n",
    "        # Made a custom version to normalize per attribute\n",
    "        # return observation.to_vect() - like object scaled accordingly\n",
    "        li_vect=  []\n",
    "        for el in observation.attr_list_vect:\n",
    "            v = observation._get_array_from_attr_name(el).astype(np.float)\n",
    "            v_fix = np.nan_to_num(v)\n",
    "            v_norm = np.linalg.norm(v_fix)\n",
    "            if v_norm > 1e8:\n",
    "                v_res = (v_fix / v_norm) * 10.0\n",
    "            else:\n",
    "                v_res = v_fix\n",
    "            li_vect.append(v_res)\n",
    "        return np.concatenate(li_vect)\n",
    "\n",
    "    def convert_act(self, action):\n",
    "        \"\"\"\n",
    "        calling the convert_act method of the base class.\n",
    "        This is not mandatory as this is the standard behaviour in OOP (object oriented programming)\n",
    "        \"\"\"\n",
    "        return super().convert_act(action)\n",
    "   \n",
    "    def my_act(self, state, reward, done=False):\n",
    "        \"\"\"\n",
    "        The complete implementation of the my_act function\n",
    "        \"\"\"\n",
    "        # Register current state to stacking buffer\n",
    "        self._save_current_frame(state)\n",
    "        # We need at least num frames to predict\n",
    "        if len(self.frames) < self.num_frames:\n",
    "            return 0 # Do nothing\n",
    "        # Infer with the last num_frames states\n",
    "        a, _ = self.Qmain.predict_move(np.array(self.frames))  # self.Qmain is of type 'DoubleDuelingDQN_NN' previously defined\n",
    "        return a\n",
    "```\n",
    "\n",
    "#### b) Training the model\n",
    "\n",
    "Now we can define the agent and train it.\n",
    "\n",
    "To that extent, we will use the \"train\" method provided in the `l2rpn_baselines` repository.\n",
    "\n",
    "**NB** The code below can take a few minutes to run. It's training a Deep Reinforcement Learning Agent after all. It this takes too long on your machine, you can always decrease the \"nb_frame\", and set it to 1000 for example. In this case, the Agent will probably not be really good.\n",
    "\n",
    "**NB** It would take much longer to train a good agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/benjamin/Documents/grid2op_dev/getting_started/grid2op/MakeEnv/Make.py:248: UserWarning: You are using a development environment. This environment is not intended for training agents.\n",
      "  warnings.warn(_MAKE_DEV_ENV_WARN)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step [0] -- Random [0.99]\n",
      "65 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [1054.1907958984375]\n",
      "25 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.445068359375]\n",
      "70 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [1] steps\n",
      "Total reward [-20.0]\n",
      "67 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5392.4451904296875]\n",
      "81 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [8671.215087890625]\n",
      "77 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1044.7841796875]\n",
      "Survived [6] steps\n",
      "Total reward [6550.37548828125]\n",
      "Survived [1] steps\n",
      "Total reward [1068.2581787109375]\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2152.339599609375]\n",
      "Survived [3] steps\n",
      "Total reward [3208.7918701171875]\n",
      "Survived [1] steps\n",
      "Total reward [1076.9752197265625]\n",
      "38 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "33 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2120.3916015625]\n",
      "69 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "24 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [6478.8660888671875]\n",
      "Survived [1] steps\n",
      "Total reward [1065.806396484375]\n",
      "Survived [1] steps\n",
      "Total reward [1076.51611328125]\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3196.2452392578125]\n",
      "Survived [3] steps\n",
      "Total reward [3250.6380615234375]\n",
      "22 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "110 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5364.99755859375]\n",
      "Survived [3] steps\n",
      "Total reward [3234.0718994140625]\n",
      "78 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "28 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4285.1968994140625]\n",
      "Survived [2] steps\n",
      "Total reward [2162.0804443359375]\n",
      "91 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [4287.4119873046875]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0350341796875]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False,  True,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "55 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7596.5767822265625]\n",
      "36 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "56 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False,  True,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [1057.4854736328125]\n",
      "Survived [2] steps\n",
      "Total reward [2134.5015869140625]\n",
      "89 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5462.8248291015625]\n",
      "Survived [2] steps\n",
      "Total reward [2129.141845703125]\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3253.2606201171875]\n",
      "Survived [2] steps\n",
      "Total reward [2133.29052734375]\n",
      "25 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "72 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "65 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [9] steps\n",
      "Total reward [6549.60400390625]\n",
      "Survived [4] steps\n",
      "Total reward [4265.5537109375]\n",
      "27 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False,  True,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [9] steps\n",
      "Total reward [8831.083740234375]\n",
      "Survived [2] steps\n",
      "Total reward [2134.2430419921875]\n",
      "Survived [1] steps\n",
      "Total reward [1077.5989990234375]\n",
      "44 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.134521484375]\n",
      "Survived [3] steps\n",
      "Total reward [3253.5682373046875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [1] steps\n",
      "Total reward [1077.9427490234375]\n",
      "Survived [4] steps\n",
      "Total reward [4272.336181640625]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0205078125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "70 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "71 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [6570.5194091796875]\n",
      "21 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5388.7926025390625]\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3243.1876220703125]\n",
      "Survived [1] steps\n",
      "Total reward [1068.5465087890625]\n",
      "Survived [2] steps\n",
      "Total reward [2162.53857421875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [1] steps\n",
      "Total reward [1072.1573486328125]\n",
      "25 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3213.531494140625]\n",
      "89 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5377.73876953125]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [2] steps\n",
      "Total reward [2134.0001220703125]\n",
      "Survived [3] steps\n",
      "Total reward [3251.582275390625]\n",
      "93 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "61 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [11] steps\n",
      "Total reward [9776.573364257812]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "58 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [5448.73876953125]\n",
      "loss = 1000.0\n",
      "98 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [6480.74462890625]\n",
      "71 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4337.0843505859375]\n",
      "Survived [6] steps\n",
      "Total reward [6395.336669921875]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0281982421875]\n",
      "37 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "50 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [2108.2894287109375]\n",
      "Survived [2] steps\n",
      "Total reward [2158.32080078125]\n",
      "Survived [4] steps\n",
      "Total reward [4296.619873046875]\n",
      "96 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1064.3912353515625]\n",
      "Survived [3] steps\n",
      "Total reward [3202.86083984375]\n",
      "Survived [2] steps\n",
      "Total reward [2159.319091796875]\n",
      "Survived [5] steps\n",
      "Total reward [5378.964599609375]\n",
      "Survived [3] steps\n",
      "Total reward [3251.708740234375]\n",
      "Survived [4] steps\n",
      "Total reward [4213.2039794921875]\n",
      "Survived [2] steps\n",
      "Total reward [2154.6038818359375]\n",
      "Survived [1] steps\n",
      "Total reward [1065.806396484375]\n",
      "Survived [2] steps\n",
      "Total reward [2163.2872314453125]\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "72 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5406.4141845703125]\n",
      "56 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2155.5777587890625]\n",
      "33 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2118.261474609375]\n",
      "47 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5460.7537841796875]\n",
      "Survived [5] steps\n",
      "Total reward [5367.1573486328125]\n",
      "Survived [4] steps\n",
      "Total reward [4351.2191162109375]\n",
      "Survived [1] steps\n",
      "Total reward [1067.76806640625]\n",
      "Survived [2] steps\n",
      "Total reward [2159.9254150390625]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "40 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3220.1846923828125]\n",
      "Survived [1] steps\n",
      "Total reward [1077.8953857421875]\n",
      "Survived [2] steps\n",
      "Total reward [2134.3626708984375]\n",
      "Survived [3] steps\n",
      "Total reward [3247.0164794921875]\n",
      "65 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2118.4864501953125]\n",
      "loss = 1.2650325\n",
      "Survived [2] steps\n",
      "Total reward [2161.9239501953125]\n",
      "Survived [1] steps\n",
      "Total reward [1068.3087158203125]\n",
      "35 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "46 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "43 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [13] steps\n",
      "Total reward [10911.33154296875]\n",
      "Survived [5] steps\n",
      "Total reward [5378.15087890625]\n",
      "25 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1063.2584228515625]\n",
      "Survived [3] steps\n",
      "Total reward [3206.488525390625]\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2157.2525634765625]\n",
      "78 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False,  True, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "22 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "67 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [14] steps\n",
      "Total reward [11985.861450195312]\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "66 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5425.99951171875]\n",
      "51 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2130.331298828125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [1] steps\n",
      "Total reward [1067.7728271484375]\n",
      "Survived [4] steps\n",
      "Total reward [4350.2462158203125]\n",
      "67 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "68 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "27 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [14] steps\n",
      "Total reward [12030.788208007812]\n",
      "Survived [2] steps\n",
      "Total reward [2163.0028076171875]\n",
      "32 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.558837890625]\n",
      "60 {'disc_lines': array([False, False, False, False, False,  True, False, False, False,\n",
      "       False, False, False, False, False,  True, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3250.0831298828125]\n",
      "85 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7570.2608642578125]\n",
      "65 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [2160.118408203125]\n",
      "41 {'disc_lines': array([False,  True, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3195.6221923828125]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0281982421875]\n",
      "Survived [4] steps\n",
      "Total reward [4294.71630859375]\n",
      "59 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4266.76220703125]\n",
      "loss = 0.93467665\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.0787353515625]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "91 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [6587.4456787109375]\n",
      "Survived [7] steps\n",
      "Total reward [7434.1304931640625]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2134.1895751953125]\n",
      "Survived [3] steps\n",
      "Total reward [3251.0855712890625]\n",
      "63 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3218.6656494140625]\n",
      "Survived [1] steps\n",
      "Total reward [1078.121826171875]\n",
      "Survived [2] steps\n",
      "Total reward [2134.676513671875]\n",
      "Survived [2] steps\n",
      "Total reward [2163.11962890625]\n",
      "81 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "32 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3189.1473388671875]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0765380859375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2110.2872314453125]\n",
      "23 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "86 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [6531.9892578125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [1] steps\n",
      "Total reward [1078.09765625]\n",
      "50 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [8743.67822265625]\n",
      "46 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [1] steps\n",
      "Total reward [-20.0]\n",
      "Survived [1] steps\n",
      "Total reward [1068.6573486328125]\n",
      "Survived [1] steps\n",
      "Total reward [1072.1573486328125]\n",
      "40 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3199.134765625]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2134.0382080078125]\n",
      "86 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [-30.0]\n",
      "31 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7586.9600830078125]\n",
      "Survived [2] steps\n",
      "Total reward [2163.2286376953125]\n",
      "Survived [3] steps\n",
      "Total reward [3205.1632080078125]\n",
      "Survived [2] steps\n",
      "Total reward [2161.3433837890625]\n",
      "92 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "72 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "59 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "98 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "loss = 0.89233637\n",
      "Survived [15] steps\n",
      "Total reward [13260.33154296875]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [1] steps\n",
      "Total reward [-20.0]\n",
      "Survived [1] steps\n",
      "Total reward [1068.4443359375]\n",
      "Survived [2] steps\n",
      "Total reward [2161.15771484375]\n",
      "Survived [1] steps\n",
      "Total reward [1067.6485595703125]\n",
      "96 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "86 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5409.4866943359375]\n",
      "93 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [1] steps\n",
      "Total reward [-20.0]\n",
      "Survived [5] steps\n",
      "Total reward [5448.248046875]\n",
      "Survived [2] steps\n",
      "Total reward [2123.3555908203125]\n",
      "79 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "74 {'disc_lines': array([False, False, False, False, False,  True, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3233.3885498046875]\n",
      "91 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [9655.288818359375]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0765380859375]\n",
      "Survived [2] steps\n",
      "Total reward [2134.5340576171875]\n",
      "38 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "59 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [4246.1068115234375]\n",
      "Survived [2] steps\n",
      "Total reward [2130.529052734375]\n",
      "Survived [1] steps\n",
      "Total reward [1077.6602783203125]\n",
      "Survived [1] steps\n",
      "Total reward [1068.2078857421875]\n",
      "76 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "78 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2145.5184326171875]\n",
      "Survived [2] steps\n",
      "Total reward [2133.6778564453125]\n",
      "100 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "52 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3250.20068359375]\n",
      "89 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "42 {'disc_lines': array([False, False, False, False, False,  True, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "61 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [5348.52978515625]\n",
      "29 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2156.39111328125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2163.226806640625]\n",
      "Survived [1] steps\n",
      "Total reward [1068.6549072265625]\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "64 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1066.00048828125]\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [2] steps\n",
      "Total reward [1045.40283203125]\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1067.814697265625]\n",
      "65 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "36 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "66 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "56 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [3227.4918212890625]\n",
      "94 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "58 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5447.85791015625]\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "loss = 0.97721255\n",
      "Survived [5] steps\n",
      "Total reward [4286.1351318359375]\n",
      "Survived [2] steps\n",
      "Total reward [2163.459228515625]\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [6485.347900390625]\n",
      "52 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3250.7252197265625]\n",
      "Survived [2] steps\n",
      "Total reward [2133.70654296875]\n",
      "Survived [3] steps\n",
      "Total reward [3248.614013671875]\n",
      "51 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [3205.997802734375]\n",
      "Survived [4] steps\n",
      "Total reward [4348.015625]\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "43 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "37 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [6545.6290283203125]\n",
      "Survived [3] steps\n",
      "Total reward [3251.2073974609375]\n",
      "36 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3211.4542236328125]\n",
      "58 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "53 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "45 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "83 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "88 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [1041.850830078125]\n",
      "Survived [7] steps\n",
      "Total reward [7496.456298828125]\n",
      "71 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "69 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2142.1182861328125]\n",
      "74 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2122.9814453125]\n",
      "Survived [8] steps\n",
      "Total reward [8805.954345703125]\n",
      "43 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2123.4952392578125]\n",
      "79 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [5490.162353515625]\n",
      "Survived [2] steps\n",
      "Total reward [2134.4056396484375]\n",
      "Survived [3] steps\n",
      "Total reward [3252.7733154296875]\n",
      "42 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4297.5262451171875]\n",
      "70 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4318.70654296875]\n",
      "54 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4223.0950927734375]\n",
      "loss = 1.0295558\n",
      "Survived [6] steps\n",
      "Total reward [6559.6904296875]\n",
      "92 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "41 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5333.0728759765625]\n",
      "62 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7683.4737548828125]\n",
      "38 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4286.2938232421875]\n",
      "76 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False,  True, False, False, False,\n",
      "       False, False, False, False, False,  True, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "64 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [2132.9991455078125]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "37 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [1] steps\n",
      "Total reward [-20.0]\n",
      "88 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3199.5521240234375]\n",
      "46 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3206.9664306640625]\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5415.6025390625]\n",
      "Survived [3] steps\n",
      "Total reward [3208.05908203125]\n",
      "Survived [5] steps\n",
      "Total reward [5440.004150390625]\n",
      "Survived [2] steps\n",
      "Total reward [2134.5023193359375]\n",
      "Survived [4] steps\n",
      "Total reward [4351.4698486328125]\n",
      "Survived [4] steps\n",
      "Total reward [4288.7147216796875]\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [1] steps\n",
      "Total reward [-20.0]\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2118.170654296875]\n",
      "75 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "41 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1067.983154296875]\n",
      "Survived [2] steps\n",
      "Total reward [2133.6697998046875]\n",
      "Survived [2] steps\n",
      "Total reward [2162.2008056640625]\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5381.0997314453125]\n",
      "Survived [4] steps\n",
      "Total reward [4349.6572265625]\n",
      "Step [1000] -- Random [0.9430048473807477]\n",
      "26 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Successfully saved model at: saved_agent_DDDQN_1000/test_agent.h5\n",
      "Survived [3] steps\n",
      "Total reward [2123.434814453125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "46 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5389.7003173828125]\n",
      "Survived [2] steps\n",
      "Total reward [2162.143310546875]\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "35 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3170.8035888671875]\n",
      "Survived [3] steps\n",
      "Total reward [3253.362548828125]\n",
      "loss = 0.7338074\n",
      "Survived [7] steps\n",
      "Total reward [7491.163330078125]\n",
      "Survived [2] steps\n",
      "Total reward [2162.9979248046875]\n",
      "79 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2120.690673828125]\n",
      "92 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5458.8182373046875]\n",
      "92 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5398.1055908203125]\n",
      "Survived [1] steps\n",
      "Total reward [1078.1407470703125]\n",
      "87 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [2130.6939697265625]\n",
      "44 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [6] steps\n",
      "Total reward [5473.4935302734375]\n",
      "71 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "22 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3192.4378662109375]\n",
      "76 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "77 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "28 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "76 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [13] steps\n",
      "Total reward [8894.699829101562]\n",
      "61 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "75 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [9] steps\n",
      "Total reward [6524.48193359375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "42 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1045.866943359375]\n",
      "Survived [2] steps\n",
      "Total reward [2162.4342041015625]\n",
      "91 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "79 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4254.627685546875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "90 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "79 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [11] steps\n",
      "Total reward [8756.936767578125]\n",
      "Survived [3] steps\n",
      "Total reward [3253.561279296875]\n",
      "71 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5363.73681640625]\n",
      "22 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5470.5118408203125]\n",
      "Survived [9] steps\n",
      "Total reward [9610.116821289062]\n",
      "Survived [2] steps\n",
      "Total reward [2160.168701171875]\n",
      "loss = 0.64850456\n",
      "72 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.6573486328125]\n",
      "Survived [8] steps\n",
      "Total reward [8775.410888671875]\n",
      "Survived [4] steps\n",
      "Total reward [4291.6058349609375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [3] steps\n",
      "Total reward [3208.7139892578125]\n",
      "97 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "85 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5360.64453125]\n",
      "37 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "72 {'disc_lines': array([False, False, False, False, False,  True, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5392.03759765625]\n",
      "22 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "59 {'disc_lines': array([False, False, False, False, False,  True, False, False, False,\n",
      "       False, False, False, False, False,  True, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [1058.1507568359375]\n",
      "50 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5343.574462890625]\n",
      "Survived [3] steps\n",
      "Total reward [3252.1175537109375]\n",
      "26 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "36 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "90 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [9] steps\n",
      "Total reward [5403.792724609375]\n",
      "Survived [6] steps\n",
      "Total reward [6566.1114501953125]\n",
      "Survived [3] steps\n",
      "Total reward [3188.3331298828125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [6] steps\n",
      "Total reward [6470.02880859375]\n",
      "69 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2156.411376953125]\n",
      "Survived [3] steps\n",
      "Total reward [3185.9384765625]\n",
      "Survived [6] steps\n",
      "Total reward [6566.828125]\n",
      "53 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "83 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "84 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "50 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Successfully saved model at: saved_agent_DDDQN_1000/test_agent.h5\n"
     ]
    }
   ],
   "source": [
    "# create an environment\n",
    "env = make(env_name, test=True)  \n",
    "# don't forget to set \"test=False\" (or remove it, as False is the default value) for \"real\" training\n",
    "\n",
    "# import the train function and train your agent\n",
    "from l2rpn_baselines.DoubleDuelingDQN import train\n",
    "agent_name = \"test_agent\"\n",
    "save_path = \"saved_agent_DDDQN_{}\".format(train_iter)\n",
    "train(env,\n",
    "      name=agent_name,\n",
    "      iterations=train_iter,\n",
    "      save_path=save_path,\n",
    "      load_path=None, # put something else if you want to reload an agent instead of creating a new one\n",
    "      logs_path=\"tf_logs_DDDQN\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logs are saved in the \"tf_logs_DDDQN\" log repository. To watch what happens during training, you can type the command (from a bash command line for example):\n",
    "```\n",
    "tensorboard --logdir='tf_logs_DDDQN'\n",
    "```\n",
    "You can even do it while it's training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III) Evaluating the Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, it is time to test the agent that we trained.\n",
    "\n",
    "To do that, we have multiple choices.\n",
    "\n",
    "We can either re-code the \"DeepQAgent\" class to load the stored weights (that have been saved during training) when it is initialized (this is not covered in this notebook), or we can also directly specify the instance of the Agent to use in the Grid2Op Runner.\n",
    "\n",
    "Doing this is fairly simple. First, you need to specify that you won't use the \"*agentClass*\" argument by setting it to ``None``, and secondly you simply have to provide the agent instance to be used as the *agentInstance* argument.\n",
    "\n",
    "**NB** If you don't do that, the Runner will be created (the constructor will raise an exception). If you choose to feed the \"*agentClass*\" argument with a class, your agent will be re-instanciated from scratch with this class. If you do re-instanciate your agent and if it is not planned in the class to load pre-trained weights, then **the agent will not be pre-trained** and will be unlikely to perform well on the task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### III.A) Evaluate the Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have \"successfully\" trained our Agent, we will evaluate it. As opposed to the training, the evaluation is done classically using a standard Runner.\n",
    "\n",
    "Note that the Runner will use a \"scoring function\" that might be different from the \"reward function\" used during training. In our case, We use the `L2RPNReward` function for both training and evaluating.\n",
    "\n",
    "In the code below, we commented on what can be different and what must be identical for training and evaluating a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Succesfully loaded network from: saved_agent_DDDQN_1000/test_agent.h5\n"
     ]
    }
   ],
   "source": [
    "from grid2op.Runner import Runner\n",
    "\n",
    "# chose a scoring function (might be different from the reward you use to train your agent)\n",
    "from grid2op.Reward import L2RPNReward\n",
    "scoring_function = L2RPNReward\n",
    "\n",
    "# load your agent\n",
    "from l2rpn_baselines.DoubleDuelingDQN import DoubleDuelingDQN\n",
    "my_agent = DoubleDuelingDQN(env.observation_space, env.action_space)\n",
    "my_agent.load(os.path.join(save_path, \"{}.h5\".format(agent_name)))\n",
    "\n",
    "# here we do that to limit the time take, and will only assess the performance on \"max_iter\" iteration\n",
    "dict_params = env.get_params_for_runner()\n",
    "dict_params[\"gridStateclass_kwargs\"][\"max_iter\"] =  max_iter\n",
    "# make a runner from an intialized environment\n",
    "runner = Runner(**dict_params, agentClass=None, agentInstance=my_agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will run the Agent and save the results. In order to do that, we specify the `path_save` argument. This path allows you to save lots of insights about your Agent's behaviour. Details of all of the information contained in this output are listed in the documentation [here](file:///home/donnotben/Documents/Grid2Op/documentation/html/runner.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results for the trained agent are:\n",
      "\tFor chronics located at 0\n",
      "\t\t - total score: 41517.792969\n",
      "\t\t - number of time steps completed: 100 / 100\n",
      "\tFor chronics located at 1\n",
      "\t\t - total score: 42068.773438\n",
      "\t\t - number of time steps completed: 100 / 100\n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "path_save=\"trained_agent_log\"\n",
    "\n",
    "# delete the previous stored results\n",
    "if os.path.exists(path_save):\n",
    "    shutil.rmtree(path_save)\n",
    "\n",
    "# run the episode\n",
    "res = runner.run(nb_episode=2, path_save=path_save)\n",
    "print(\"The results for the trained agent are:\")\n",
    "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics located at {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - total score: {:.6f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### III.B) Inspect the Agent "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please refer to the official documentation for more information about the contents of the directory where the data is saved. Note that saving the information is triggered by the \"path_save\" argument of the \"runner.run\" function.\n",
    "\n",
    "The information contained in this output will be saved in a structured way and includes :\n",
    "For each episode :\n",
    "  - \"episode_meta.json\": json file that represents some meta information about:\n",
    "\n",
    "    - \"backend_type\": the name of the `grid2op.Backend` class used\n",
    "    - \"chronics_max_timestep\": the **maximum** number of timesteps for the chronics used\n",
    "    - \"chronics_path\": the path where the temporal data (chronics) are located\n",
    "    - \"env_type\": the name of the `grid2op.Environment` class used.\n",
    "    - \"grid_path\": the path where the powergrid has been loaded from\n",
    "\n",
    "  - \"episode_times.json\": json file that gives some information about the total time spent in multiple parts of the runner, mainly the\n",
    "    `grid2op.Agent` (and especially its method `grid2op.Agent.act`) and the\n",
    "    `grid2op.Environment`\n",
    "\n",
    "  - \"_parameters.json\": json representation of the `grid2op.Parameters` used for this episode\n",
    "  - \"rewards.npy\": numpy 1d-array giving the rewards at each time step. We adopted the convention that the stored\n",
    "    reward at index `i` is the one observed by the agent at time `i` and **NOT** the reward sent by the\n",
    "    `grid2op.Environment` after the action has been taken.\n",
    "  - \"exec_times.npy\": numpy 1d-array giving the execution time for each time step in the episode\n",
    "  - \"actions.npy\": numpy 2d-array giving the actions that have been taken by the `grid2op.Agent`. At row `i` of \"actions.npy\" is a\n",
    "    vectorized representation of the action performed by the agent at timestep `i` *ie.* **after** having observed\n",
    "    the observation present at row `i` of \"observation.npy\" and the reward showed in row `i` of \"rewards.npy\".\n",
    "  - \"disc_lines.npy\": numpy 2d-array that tells which lines have been disconnected during the simulation of the cascading failure at each\n",
    "    time step. The same convention has been adopted for \"rewards.npy\". This means that the powerlines are\n",
    "    disconnected when the `grid2op.Agent` takes the `grid2op.Action` at time step `i`.\n",
    "  - \"observations.npy\": numpy 2d-array representing the `grid2op.Observation` at the disposal of the\n",
    "    `grid2op.Agent` when he took his action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can first look at the repository were the data is stored:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['dict_env_modification_space.json',\n",
       " '1',\n",
       " 'dict_action_space.json',\n",
       " 'dict_observation_space.json',\n",
       " '0']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.listdir(path_save)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see there are 2 folders, each corresponding to a chronics.\n",
    "There are also additional json files.\n",
    "\n",
    "Now let's see what is inside one of these folders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['episode_meta.json',\n",
       " 'disc_lines_cascading_failure.npz',\n",
       " 'actions.npz',\n",
       " 'rewards.npz',\n",
       " 'observations.npz',\n",
       " 'agent_exec_times.npz',\n",
       " 'episode_times.json',\n",
       " '_parameters.json',\n",
       " 'other_rewards.json',\n",
       " 'env_modifications.npz']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(os.path.join(path_save, \"0\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can load the actions chosen by the Agent, and have a look at them.\n",
    "\n",
    "To do that, we will load the action array and use the `action_space` function to convert it back to `Action` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grid2op.Episode import EpisodeData\n",
    "this_episode = EpisodeData.from_disk(path_save, name=\"0\")\n",
    "all_actions = this_episode.get_actions()\n",
    "li_actions = []\n",
    "for i in range(all_actions.shape[0]):\n",
    "    try:\n",
    "        tmp = runner.env.action_space.from_vect(all_actions[i,:])\n",
    "        li_actions.append(tmp)\n",
    "    except:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This allows us to have a deeper look at the actions, and their effects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will inspect the actions that has been taken by the agent :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total reconnected lines : 0\n",
      "Total disconnected lines : 0\n"
     ]
    }
   ],
   "source": [
    "line_disc = 0\n",
    "line_reco = 0\n",
    "for act in li_actions:\n",
    "    dict_ = act.as_dict()\n",
    "    if \"set_line_status\" in dict_:\n",
    "        line_reco +=  dict_[\"set_line_status\"][\"nb_connected\"]\n",
    "        line_disc +=  dict_[\"set_line_status\"][\"nb_disconnected\"]\n",
    "print(f'Total reconnected lines : {line_reco}')\n",
    "print(f'Total disconnected lines : {line_disc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, during this episode, our agent never tries to disconnect or reconnect a line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also analyse the observations of the recorded episode :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of disconnected powerlines cumulated over all the timesteps : 0\n"
     ]
    }
   ],
   "source": [
    "all_observations = this_episode.get_observations()\n",
    "li_observations = []\n",
    "nb_real_disc = 0\n",
    "for i in range(all_observations.shape[0]):\n",
    "    try:\n",
    "        tmp = runner.env.observation_space.from_vect(all_observations[i,:])\n",
    "        li_observations.append(tmp)\n",
    "        nb_real_disc += (np.sum(tmp.line_status == False))\n",
    "    except:\n",
    "        break\n",
    "print(f'Total number of disconnected powerlines cumulated over all the timesteps : {nb_real_disc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the kind of actions that the agent chose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The agent did 2 different valid actions:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "actions_count = {}\n",
    "for act in li_actions:\n",
    "    act_as_vect = tuple(act.to_vect())\n",
    "    if not act_as_vect in actions_count:\n",
    "        actions_count[act_as_vect] = 0\n",
    "    actions_count[act_as_vect] += 1\n",
    "print(\"The agent did {} different valid actions:\\n\".format(len(actions_count)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actions chosen by the agent were :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - NOT perform any redispatching action\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - NOT force any particular bus configuration\n",
      "This action will:\n",
      "\t - NOT change anything to the injections\n",
      "\t - Redispatch gen_0_4 of -7.5\n",
      "\t - NOT force any line status\n",
      "\t - NOT switch any line status\n",
      "\t - NOT switch anything in the topology\n",
      "\t - NOT force any particular bus configuration\n"
     ]
    }
   ],
   "source": [
    "all_act = np.array(list(actions_count.keys()))\n",
    "for act in all_act:\n",
    "    print(runner.env.action_space.from_vect(act))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV) Improve your Agent "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw, the agent we developped was not really interesting. To improve it, we could think about:\n",
    "\n",
    "- a better encoding of the observation. For now everything, the whole observation is fed to the neural network, and we do not try to modify it at all. This is a real problem for learning algorithms.\n",
    "- a better neural network architecture (as said, we didn't pay any attention to it in our model)\n",
    "- training it for a longer time\n",
    "- adapting the learning rate and all the hyper-parameters of the learning algorithm.\n",
    "- etc.\n",
    "\n",
    "In this notebook, we will focus on changing the observation representation, by only feeding to the agent a part of the available information.\n",
    "\n",
    "To do so, the only thing that we need to do is to modify the way the observation is converted in the `convert_obs` method, and that is it. Nothing else needs to be changed. Here for example, we could think of only using the flow ratio (i.e., the current flow divided by the thermal limit, named rho) instead of feeding the whole observation to the agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DoubleDuelingDQN_Improved(DoubleDuelingDQN):\n",
    "    def convert_obs(self, observation):\n",
    "        \"\"\"\n",
    "        And by just changing that, i can change what is fed to the neural network :-)\n",
    "        \n",
    "        NB: i need however to tell in the initialization of the neural network the changes I made...\n",
    "        \"\"\"\n",
    "        return observation.rho\n",
    "    \n",
    "    def __init__(self,\n",
    "                 observation_space,\n",
    "                 action_space,\n",
    "                 name=__name__,\n",
    "                 num_frames=4,\n",
    "                 is_training=False,\n",
    "                 batch_size=32,\n",
    "                 lr=1e-5):\n",
    "        \"\"\"\n",
    "        We have changed the size of the observation, so we need to re create another neural network with\n",
    "        the proper input size. \n",
    "        That is why we need to change this.\n",
    "        \"\"\"\n",
    "        # Call parent constructor\n",
    "        DoubleDuelingDQN.__init__(self,\n",
    "                                  observation_space=observation_space,\n",
    "                                  action_space=action_space,\n",
    "                                  name=name,\n",
    "                                  num_frames=num_frames,\n",
    "                                  is_training=is_training,\n",
    "                                  batch_size=batch_size,\n",
    "                                  lr=lr)\n",
    "        \n",
    "        # import some constant and the class for this baseline\n",
    "        from l2rpn_baselines.DoubleDuelingDQN.DoubleDuelingDQN_NN import DoubleDuelingDQN_NN\n",
    "        from l2rpn_baselines.DoubleDuelingDQN.DoubleDuelingDQN import LR_DECAY_STEPS, LR_DECAY_RATE\n",
    "        \n",
    "        # Compute dimensions from intial spaces\n",
    "        self.observation_size = self.obs_space.n_line\n",
    "\n",
    "        # Load network graph\n",
    "        self.Qmain = DoubleDuelingDQN_NN(self.action_size,\n",
    "                                         self.observation_size,\n",
    "                                         num_frames = self.num_frames,\n",
    "                                         learning_rate = self.lr,\n",
    "                                         learning_rate_decay_steps = LR_DECAY_STEPS,\n",
    "                                         learning_rate_decay_rate = LR_DECAY_RATE)\n",
    "        \n",
    "        # Setup training vars if needed\n",
    "        if self.is_training:\n",
    "            self._init_training()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can reuse the generic method provided by l2rpn_baselines to train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step [0] -- Random [0.99]\n",
      "Survived [5] steps\n",
      "Total reward [5447.2547607421875]\n",
      "32 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3216.988525390625]\n",
      "84 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5470.306396484375]\n",
      "Survived [5] steps\n",
      "Total reward [5380.9202880859375]\n",
      "36 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "65 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4275.7816162109375]\n",
      "85 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5404.2862548828125]\n",
      "Survived [4] steps\n",
      "Total reward [4349.153076171875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2139.8360595703125]\n",
      "Survived [2] steps\n",
      "Total reward [2128.30615234375]\n",
      "93 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3213.2412109375]\n",
      "Survived [3] steps\n",
      "Total reward [3125.3204345703125]\n",
      "58 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "86 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "61 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "54 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [5472.8387451171875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "44 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "27 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [9] steps\n",
      "Total reward [6419.149169921875]\n",
      "Survived [7] steps\n",
      "Total reward [7581.1126708984375]\n",
      "95 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "43 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "76 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "24 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "22 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [3270.89306640625]\n",
      "74 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.735595703125]\n",
      "91 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "21 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4342.416259765625]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [3] steps\n",
      "Total reward [3207.1396484375]\n",
      "43 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7684.775146484375]\n",
      "47 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "27 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "45 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "77 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [3187.4453125]\n",
      "72 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "loss = 549.62305\n",
      "Survived [4] steps\n",
      "Total reward [3252.4393310546875]\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "80 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [2119.525634765625]\n",
      "Survived [1] steps\n",
      "Total reward [1078.03271484375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [1] steps\n",
      "Total reward [1077.0946044921875]\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3206.1614990234375]\n",
      "43 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2155.7093505859375]\n",
      "83 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5370.4114990234375]\n",
      "Survived [1] steps\n",
      "Total reward [1077.9410400390625]\n",
      "Survived [1] steps\n",
      "Total reward [1061.6875]\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "28 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4350.8052978515625]\n",
      "Survived [1] steps\n",
      "Total reward [1068.6011962890625]\n",
      "28 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False,  True, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2140.3885498046875]\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "64 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5401.374267578125]\n",
      "Survived [5] steps\n",
      "Total reward [5452.2530517578125]\n",
      "85 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1053.67333984375]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0205078125]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [4] steps\n",
      "Total reward [4295.6754150390625]\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3246.2843017578125]\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "62 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "70 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [7604.53271484375]\n",
      "41 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "95 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5456.4700927734375]\n",
      "33 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "85 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "46 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [12] steps\n",
      "Total reward [9859.084838867188]\n",
      "Survived [2] steps\n",
      "Total reward [2162.9840087890625]\n",
      "Survived [2] steps\n",
      "Total reward [2120.7664794921875]\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7608.9593505859375]\n",
      "63 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2118.75830078125]\n",
      "loss = 5.7135715\n",
      "Survived [1] steps\n",
      "Total reward [1078.138427734375]\n",
      "45 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4286.1954345703125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [3] steps\n",
      "Total reward [3206.850341796875]\n",
      "32 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1067.9581298828125]\n",
      "78 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2131.2760009765625]\n",
      "42 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "70 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2152.5272216796875]\n",
      "66 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7591.3861083984375]\n",
      "81 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7691.6976318359375]\n",
      "90 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "52 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4286.4683837890625]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5402.93505859375]\n",
      "Survived [2] steps\n",
      "Total reward [2133.382568359375]\n",
      "Survived [2] steps\n",
      "Total reward [2163.4803466796875]\n",
      "88 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "76 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4296.4527587890625]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [4] steps\n",
      "Total reward [4251.0028076171875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "69 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "52 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7682.350830078125]\n",
      "66 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5397.57763671875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2134.30712890625]\n",
      "41 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "56 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "100 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "31 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [5477.4190673828125]\n",
      "21 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "86 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [8572.2451171875]\n",
      "Survived [4] steps\n",
      "Total reward [4346.0970458984375]\n",
      "Survived [1] steps\n",
      "Total reward [1068.2777099609375]\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "loss = 5.5138416\n",
      "Survived [2] steps\n",
      "Total reward [1065.356201171875]\n",
      "32 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [9822.246704101562]\n",
      "54 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [8] steps\n",
      "Total reward [7656.605712890625]\n",
      "Survived [1] steps\n",
      "Total reward [1068.5875244140625]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2132.947265625]\n",
      "Survived [2] steps\n",
      "Total reward [2159.0985107421875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [5] steps\n",
      "Total reward [5415.718505859375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "52 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [3252.9639892578125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "40 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1066.4849853515625]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5394.5157470703125]\n",
      "97 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "59 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False,  True, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [11] steps\n",
      "Total reward [8831.709716796875]\n",
      "Survived [3] steps\n",
      "Total reward [3201.95361328125]\n",
      "26 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2155.6617431640625]\n",
      "66 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "24 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3216.906005859375]\n",
      "Survived [2] steps\n",
      "Total reward [2163.0545654296875]\n",
      "64 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "67 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2089.5565185546875]\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2150.997314453125]\n",
      "Survived [2] steps\n",
      "Total reward [2131.6165771484375]\n",
      "35 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [2163.08984375]\n",
      "27 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "95 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5380.3294677734375]\n",
      "Survived [3] steps\n",
      "Total reward [3246.218017578125]\n",
      "46 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4285.9005126953125]\n",
      "44 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "59 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4326.5576171875]\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "88 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "loss = 5.7085476\n",
      "Survived [6] steps\n",
      "Total reward [4285.4490966796875]\n",
      "Survived [2] steps\n",
      "Total reward [2161.9312744140625]\n",
      "87 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [9] steps\n",
      "Total reward [8715.129272460938]\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3254.991455078125]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.17333984375]\n",
      "51 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3252.7314453125]\n",
      "Survived [2] steps\n",
      "Total reward [2133.1719970703125]\n",
      "29 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "47 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [12] steps\n",
      "Total reward [11165.73486328125]\n",
      "75 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "85 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [4249.6219482421875]\n",
      "Survived [3] steps\n",
      "Total reward [3207.1534423828125]\n",
      "Survived [9] steps\n",
      "Total reward [9815.837890625]\n",
      "80 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "36 {'disc_lines': array([False, False, False, False, False,  True, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "43 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [4317.823974609375]\n",
      "Survived [2] steps\n",
      "Total reward [2133.80615234375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [4] steps\n",
      "Total reward [4347.6173095703125]\n",
      "Survived [1] steps\n",
      "Total reward [1068.5543212890625]\n",
      "29 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [9978.790771484375]\n",
      "23 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2092.926025390625]\n",
      "66 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "91 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [1058.9713134765625]\n",
      "96 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3208.377197265625]\n",
      "70 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5414.6861572265625]\n",
      "84 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "38 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "95 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [7683.4754638671875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "loss = 5.867881\n",
      "Survived [1] steps\n",
      "Total reward [1068.445068359375]\n",
      "Survived [2] steps\n",
      "Total reward [2163.2646484375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "76 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "78 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5468.2979736328125]\n",
      "30 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "50 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [8698.8662109375]\n",
      "Survived [1] steps\n",
      "Total reward [1072.1573486328125]\n",
      "Survived [5] steps\n",
      "Total reward [5380.174560546875]\n",
      "Survived [2] steps\n",
      "Total reward [2152.569580078125]\n",
      "58 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3197.6474609375]\n",
      "Survived [4] steps\n",
      "Total reward [4350.26220703125]\n",
      "Survived [2] steps\n",
      "Total reward [2134.212158203125]\n",
      "Survived [3] steps\n",
      "Total reward [3229.5057373046875]\n",
      "95 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [4296.0013427734375]\n",
      "Survived [3] steps\n",
      "Total reward [3189.661865234375]\n",
      "Survived [5] steps\n",
      "Total reward [5375.5506591796875]\n",
      "58 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "79 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "110 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [16] steps\n",
      "Total reward [13324.2919921875]\n",
      "83 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2120.48486328125]\n",
      "Survived [4] steps\n",
      "Total reward [4351.473876953125]\n",
      "Survived [3] steps\n",
      "Total reward [3205.2999267578125]\n",
      "Survived [3] steps\n",
      "Total reward [3240.354736328125]\n",
      "40 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [3219.5596923828125]\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4354.072998046875]\n",
      "69 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5335.634765625]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0638427734375]\n",
      "Survived [5] steps\n",
      "Total reward [5380.6141357421875]\n",
      "39 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "loss = 5.8966236\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [1055.3546142578125]\n",
      "Survived [1] steps\n",
      "Total reward [1068.6448974609375]\n",
      "Survived [8] steps\n",
      "Total reward [8805.762817382812]\n",
      "Survived [2] steps\n",
      "Total reward [2132.171630859375]\n",
      "Survived [3] steps\n",
      "Total reward [3248.901123046875]\n",
      "Survived [2] steps\n",
      "Total reward [2134.373779296875]\n",
      "Survived [4] steps\n",
      "Total reward [4348.080810546875]\n",
      "Survived [6] steps\n",
      "Total reward [6480.16357421875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [2] steps\n",
      "Total reward [2134.17626953125]\n",
      "58 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [6572.3677978515625]\n",
      "64 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [4293.0633544921875]\n",
      "Survived [2] steps\n",
      "Total reward [2163.3460693359375]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "38 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [6572.541015625]\n",
      "Survived [3] steps\n",
      "Total reward [3205.36474609375]\n",
      "Survived [4] steps\n",
      "Total reward [4336.66845703125]\n",
      "Survived [2] steps\n",
      "Total reward [2134.4091796875]\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [10] steps\n",
      "Total reward [9695.97900390625]\n",
      "Survived [1] steps\n",
      "Total reward [1068.6573486328125]\n",
      "Survived [1] steps\n",
      "Total reward [1078.080078125]\n",
      "Survived [4] steps\n",
      "Total reward [4295.674560546875]\n",
      "Survived [4] steps\n",
      "Total reward [4287.5382080078125]\n",
      "32 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "75 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "82 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [3203.0211181640625]\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5471.9775390625]\n",
      "77 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "99 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "55 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3209.9154052734375]\n",
      "Survived [2] steps\n",
      "Total reward [2163.792236328125]\n",
      "71 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1045.851318359375]\n",
      "loss = 5.8687263\n",
      "35 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1067.983154296875]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "41 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "42 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2145.5057373046875]\n",
      "28 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4284.32568359375]\n",
      "Survived [4] steps\n",
      "Total reward [4349.329833984375]\n",
      "77 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [1058.43603515625]\n",
      "Survived [3] steps\n",
      "Total reward [3252.26171875]\n",
      "83 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4290.8255615234375]\n",
      "88 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [6] steps\n",
      "Total reward [5458.866455078125]\n",
      "92 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4290.2779541015625]\n",
      "Survived [3] steps\n",
      "Total reward [3252.648193359375]\n",
      "77 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2133.4002685546875]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [2] steps\n",
      "Total reward [2161.0552978515625]\n",
      "Survived [4] steps\n",
      "Total reward [4294.322021484375]\n",
      "Survived [4] steps\n",
      "Total reward [4348.2237548828125]\n",
      "Survived [4] steps\n",
      "Total reward [4295.1031494140625]\n",
      "Survived [5] steps\n",
      "Total reward [5448.5179443359375]\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [4290.6700439453125]\n",
      "48 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2154.5166015625]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [4] steps\n",
      "Total reward [4295.270751953125]\n",
      "Survived [3] steps\n",
      "Total reward [3252.1734619140625]\n",
      "91 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Step [1000] -- Random [0.9295309194857401]\n",
      "Successfully saved model at: saved_agent_DDDQN_1000/test_agent2.h5\n",
      "Survived [6] steps\n",
      "Total reward [5385.0968017578125]\n",
      "60 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "45 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [5474.3079833984375]\n",
      "98 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [3] steps\n",
      "Total reward [2114.7254638671875]\n",
      "69 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "51 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "56 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [9] steps\n",
      "Total reward [6503.87890625]\n",
      "loss = 6.3086815\n",
      "48 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [2131.002197265625]\n",
      "Survived [6] steps\n",
      "Total reward [6466.460693359375]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [1] steps\n",
      "Total reward [1076.6783447265625]\n",
      "65 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "95 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3193.3489990234375]\n",
      "46 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [6557.531005859375]\n",
      "Survived [3] steps\n",
      "Total reward [3204.6378173828125]\n",
      "98 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "31 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3255.9365234375]\n",
      "71 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [11] steps\n",
      "Total reward [10845.62744140625]\n",
      "40 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "23 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "        True, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [5] steps\n",
      "Total reward [3255.5640869140625]\n",
      "Survived [1] steps\n",
      "Total reward [1067.7728271484375]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Survived [2] steps\n",
      "Total reward [2154.5325927734375]\n",
      "Survived [2] steps\n",
      "Total reward [2134.36669921875]\n",
      "Survived [2] steps\n",
      "Total reward [2161.016845703125]\n",
      "32 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "74 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [4] steps\n",
      "Total reward [2120.105712890625]\n",
      "Survived [2] steps\n",
      "Total reward [2162.016845703125]\n",
      "Survived [0] steps\n",
      "Total reward [-10.0]\n",
      "Survived [4] steps\n",
      "Total reward [4347.8699951171875]\n",
      "98 {'disc_lines': None, 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal'), Grid2OpException DivergingPowerFlow DivergingPowerFlow('Powerflow has diverged during computation.')], 'rewards': {}}\n",
      "Survived [2] steps\n",
      "Total reward [2127.703369140625]\n",
      "Survived [3] steps\n",
      "Total reward [3239.1090087890625]\n",
      "34 {'disc_lines': array([False, False, False, False, False, False, False, False, False,\n",
      "       False, False, False, False, False, False, False, False, False,\n",
      "       False, False]), 'is_illegal': True, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'exception': [Grid2OpException IllegalAction IllegalAction('BaseAction illegal')], 'rewards': {}}\n",
      "Survived [7] steps\n",
      "Total reward [6483.77587890625]\n",
      "Survived [1] steps\n",
      "Total reward [1078.0765380859375]\n",
      "Survived [1] steps\n",
      "Total reward [1068.2210693359375]\n",
      "Successfully saved model at: saved_agent_DDDQN_1000/test_agent2.h5\n"
     ]
    }
   ],
   "source": [
    "from l2rpn_baselines.utils import train_generic\n",
    "agent_name = \"test_agent2\"\n",
    "save_path = \"saved_agent_DDDQN2_{}\".format(train_iter)\n",
    "\n",
    "my_new_agent = DoubleDuelingDQN_Improved(env.observation_space,\n",
    "                                         env.action_space,\n",
    "                                         is_training=True,\n",
    "                                         name=agent_name)\n",
    "\n",
    "my_new_agent_trained = train_generic(agent=my_new_agent,\n",
    "                                     env=env,\n",
    "                                     iterations=train_iter,\n",
    "                                     save_path=\"saved_agent_DDDQN_{}\".format(train_iter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we re-use the code that we wrote earlier to assess its performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The results for the trained agent are:\n",
      "\tFor chronics located at 0\n",
      "\t\t - total score: 5379.398438\n",
      "\t\t - number of time steps completed: 6 / 100\n",
      "\tFor chronics located at 1\n",
      "\t\t - total score: 5450.483887\n",
      "\t\t - number of time steps completed: 6 / 100\n"
     ]
    }
   ],
   "source": [
    "runner2 = Runner(**dict_params, agentClass=None, agentInstance=my_new_agent_trained)\n",
    "\n",
    "# run the episode\n",
    "res = runner2.run(nb_episode=2, path_save=path_save)\n",
    "print(\"The results for the trained agent are:\")\n",
    "for _, chron_name, cum_reward, nb_time_step, max_ts in res:\n",
    "    msg_tmp = \"\\tFor chronics located at {}\\n\".format(chron_name)\n",
    "    msg_tmp += \"\\t\\t - total score: {:.6f}\\n\".format(cum_reward)\n",
    "    msg_tmp += \"\\t\\t - number of time steps completed: {:.0f} / {:.0f}\".format(nb_time_step, max_ts)\n",
    "    print(msg_tmp)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
